Comprehensive analysis of remaining performance improvements across:
- HDF5 loading (reads full dataset for shape metadata - critical fix)
- Node.process() redundant structure() call on every update
- Complex plot rendering deepcopy overhead
- Signal emission overhead (7 signals per node per update)
- largest_numtype() iterating every array element as Python objects
- Various numpy anti-patterns (np.append in loops, unnecessary copies)
- Architectural improvements (change detection, memoization)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Round 2 performance improvements:

- largest_numtype(): use numpy dtype instead of iterating every element
  as a Python object (~15,000x faster for numeric arrays)
- Node.process(): defer structure() call to only when structure changes
  (50x faster steady-state updates for large meshgrids)
- is_invalid(): skip unnecessary np.zeros allocation for non-float arrays
- guess_grid_from_sweep_direction(): convert once with np.asarray, not 4x
- remove_invalid_entries(): replace O(n^2) np.append with list+concatenate
  Also fixes crash on inhomogeneous index arrays (pre-existing bug)
- meshgrid_to_datadict/datadict_to_dataframe: ravel() instead of flatten()
- _splitComplexData(): dataclasses.replace instead of deepcopy

Adds 32 new tests (205 total passing).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename loop variable 'd' to 'diffs' to avoid shadowing the outer loop
variable from 'for d in self.dependents()'. Add explicit type annotations
for ndarray variables to satisfy mypy's type narrowing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked the full plottr pipeline (load -> DataSelector -> DataGridder
-> XYSelector) on 23 QCodes datasets of varying shapes and sizes.

Pipeline total: 1478 ms -> 1025 ms = 1.44x overall speedup.
Largest gains on big datasets: stability_diagram 1.81x, large_3d_scan 1.65x.
No regressions on any dataset.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pipeline total: 6,550 ms -> 3,465 ms = 1.89x overall speedup on 8 large
datasets (4M-point 1D, 800x800 2D, 100x100x80 3D, interrupted, multi-dep).

Consistent ~2x speedup across 1D/2D shapes, ~1.7x on 3D.
Loading times unchanged (QCodes SQLite I/O dominated).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New benchmark measures both cold start (new flowchart) and steady state
(persistent flowchart, simulating live monitoring refresh). Uses 5 repeats
with warmup, reports median.

Results on 31 datasets (23 small + 8 large):
- Large datasets: cold 1.88x, steady 1.77x faster
- Small datasets: cold 1.43x, steady 1.69x faster
- Steady-state on small data shows up to 2.11x speedup

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarks real user actions (switch dep, swap axes, toggle subtract avg,
slide dimension, toggle grid) on large datasets with per-node time breakdown.

Key findings:
- DataSelector: 10-17x faster (largest_numtype O(1), copy optimized)
- SubtractAverage: 6-29x faster (copy 15x faster, mask_invalid skips clean)
- ScaleUnits: 7-15x faster (copy 15x faster)
- XYSelector: 1.5-2.3x (cascading copy removed)
- DataGridder: 1.1x (dominated by actual gridding computation)

Action-level: toggle_subtract_avg 9-10x, swap_xy 3.3x, switch_dep 2.3x,
data_refresh 2.2x, slide_dimension 1.5-1.6x.

DataGridder is now the dominant cost (58% of pipeline) and is the next frontier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The shape-guessing algorithm (_find_switches -> find_direction_period ->
guess_grid_from_sweep_direction) was the #1 bottleneck after rounds 1-2.

Optimizations:
- Compute is_invalid() once instead of 3 times per call
- Single np.percentile([lo, hi]) call instead of two separate sorts
- Direct numpy subtraction instead of MaskedArray creation
- Vectorized boolean mask instead of Python list comprehension
- np.nanmean for NaN-safe sweep direction detection
- Cached np.std in guess_grid_from_sweep_direction

Results (800x800 = 640K pts):
- _find_switches: 80ms -> 31ms (2.6x)
- datadict_to_meshgrid: 175ms -> 71ms (2.5x)
- Cumulative pipeline speedup vs master: 2.8-3.5x

Adds 62 comprehensive gridder tests covering all GridOption paths,
edge cases, various shapes, noisy axes, incomplete data.

All 267 tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Benchmarked on production quantum device datasets:
- QDstability (223 MB, 16 deps): cold 3.56x, steady 2.93x faster
- TopogapStage2 (152 MB, 21 deps, 4D): cold 2.47x, steady 2.7x faster
- QDtuning (14 MB, 16 deps): steady 2.73x faster
- DataSelector node: 12-13x faster on these multi-dependent datasets

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Lazy snapshot loading in RunInfo:
- Snapshot tree widget items are built only when the user expands the
  'QCoDeS Snapshot' section, not on every click
- Saves ~951ms per click on datasets with 5.9 MB snapshots (3,554x faster)
- Info pane shows collapsed by default instead of expandAll()

Incremental DB refresh:
- refreshDB() now loads only new runs since last refresh using the
  start parameter of get_runs_from_db()
- Merges incremental results into existing dataframe
- First load still loads everything

All 267 tests pass. No mypy errors introduced.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add get_runs_from_db_fast() which uses load_by_id() directly per run,
bypassing the O(N^2) experiments() + data_sets() enumeration.

For 1496 runs: old approach takes 15+ minutes, new takes ~5 seconds.
Incremental refresh loads only new runs since last known run_id.

LoadDBProcess now uses get_runs_from_db_fast with start_run_id parameter
for both initial load and incremental refresh.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

RunList now shows contextual overlay messages:
- 'Loading database... (N/M datasets)' with live progress during load
- 'Select a date on the left to browse datasets.' when idle
- 'No datasets found in this database.' for empty DBs
- 'No datasets match the current filter.' when star/cross filters hide all

Progress is reported from the worker thread via progressUpdated signal,
updated every 10 datasets for smooth display without overhead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Check actual RunList widget state (topLevelItemCount) instead of
_selected_dates to decide whether to show the hint text. This handles
same-file reload, empty date selection, and filter edge cases.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Data structure and Metadata sections now collapsed by default
  (user expands what they need)
- Set ScrollPerPixel on RunInfo tree widget so tall rows (e.g., long
  exception tracebacks in metadata) can be scrolled smoothly instead
  of jumping to the next row

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a combo box in the toolbar to switch between matplotlib and
pyqtgraph backends. Default is matplotlib. The selection applies to
all newly opened plot windows. Existing windows keep their backend.

The combo box respects the --plotWidgetClass passed via constructor
(e.g., from script_pyqtgraph entrypoint).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/data/qcodes_db_overview.py:
- get_db_overview(): single SQL JOIN query for all run metadata
- Skips snapshot and run_description blobs entirely
- Reads inspectr_tag directly as a column from runs table
- 6x faster than load_by_id, ~1000x faster than experiments() enumeration
- Intended for eventual contribution to QCoDeS

Inspectr LoadDBProcess now uses SQL path by default with automatic
fallback to qcodes API (get_runs_from_db_fast) if SQL fails.

Also: default window size widened from 640x640 to 960x640.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add 'Scrollable' checkbox in both pyqtgraph and matplotlib toolbars:
- Enabled by default
- When many subplots exist, the plot area expands beyond the window
  and becomes scrollable, keeping each subplot readable
- Can be unchecked to fit everything into the visible window

PyQtGraph: min plot height reduced from 250px to 75px.
Matplotlib: canvas wraps in QScrollArea, min height set per grid row.

280 tests pass, 0 mypy errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Both backends:
- Scrollable is now OFF by default
- Added a 'px' spinbox next to the Scrollable checkbox showing the
  minimum height per subplot row (default 75px pyqtgraph, 100px mpl)
- Spinbox is only enabled when Scrollable is checked
- Minimum value is 40px
- Changing the spinbox value triggers replot

280 tests pass, 0 mypy errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Extract 'Select a date...' string into _SELECT_DATE_HINT constant
- Replace string-magic backend detection with explicit _PLOT_BACKENDS
  mapping (display name -> class)
- _backend_name_for_class() for reverse lookup
- Unknown plotWidgetClass added to combo with its class name as label
- _onBackendChanged uses the mapping instead of hardcoded imports

280 tests pass, 0 mypy errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- _split_timestamp(): proper datetime parsing instead of string slicing
  for splitting qcodes timestamp strings into date/time components.
  Applied to both get_ds_info() and _ds_to_info_dict().
- get_runs_from_db_fast(): removed unnecessary initialise_or_create_database_at
  call, use same read_only pattern as get_runs_from_db.
- qcodes_db_overview: use conn_from_dbpath_or_conn from qcodes instead of
  raw sqlite3.connect. Remove unused get_last_run_id function.
- mpl/widgets: remove dead _scrollable attribute and fix setScrollable
  which had identical code in both branches.

280 tests pass, 0 mypy errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

New module plottr/utils/latex.py using unicodeit (now a required dependency):
- Greek letters: \alpha -> α, \Omega -> Ω
- Math symbols: \hbar -> ℏ, \partial -> ∂, \int -> ∫
- Subscripts: V_{gate} -> V<sub>gate</sub> (HTML for text, Unicode for digits)
- Superscripts: x^{2} -> x² (Unicode), e^{iπ} -> e<sup>iπ</sup>
- Fractions: \frac{dI}{dV} -> dI/dV
- Square root: \sqrt{x} -> √x
- Dollar delimiters stripped

Applied to pyqtgraph axis labels in FigureMaker.formatSubPlot().
Falls through gracefully on plain text (no LaTeX = no change).

35 new tests including hypothesis property-based testing.
315 tests pass, 0 mypy errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Convert subscripts and superscripts to HTML tags BEFORE running
unicodeit, so they become <sub>11</sub> and <sup>2</sup> instead of
Unicode ₁₁ and ². HTML tags render more consistently in Qt rich text.

unicodeit still converts Greek letters and symbols inside the tags.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Plain strings with underscores (e.g. gate_voltage, channel_1_amplitude)
now pass through unchanged. Conversion only triggers when the string
contains backslash commands (\alpha), dollar delimiters ($...$), or
braced sub/superscripts (_{...}, ^{...}).

Also drops single-char bare sub/sup patterns (_x, ^x) which were too
aggressive on ordinary identifiers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Restore type: ignore[has-type] on inspectr starAction/crossAction
  (needed by PyQt5-stubs, unused with PyQt6)
- Add type: ignore[arg-type] on autonode widgetConnection.get
  (needed with PyQt6, unused with PyQt5-stubs)
- Set warn_unused_ignores = false since project targets both Qt bindings

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

qcodes moved find_scale_and_prefix from qcodes.utils.plotting to
qcodes.plotting.axis_labels. Update the import chain to try the
current location first, then the old one, then the local fallback.

Remove qcodes.utils.plotting from mypy ignore_missing_imports since
the import is now handled via try/except with proper type: ignore
annotations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Revert rettype initialization to None + assert before use (cleaner
  than pre-computing a default that changes the original semantics)
- Remove dead local fallback for qcodes < 0.21 (min supported is
  0.54.1); keep old-path fallback with version comment for < 0.46
- Restore warn_unused_ignores = true globally; add per-module override
  (warn_unused_ignores = false) only for modules with cross-Qt-backend
  type: ignore comments (inspectr, autonode, scaleunits)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Profiled plottr pipeline with actual measurement data ([redacted]
downloaded via qdwsdk. Key findings:

- is_invalid() is 44x slower than needed (a==None on numeric arrays)
- ds_to_datadict takes ~1s steady state (qcodes deserialization)
- datadict_to_meshgrid 122ms (avoidable when shape metadata exists)
- pyqtgraph eq() adds 24ms per pipeline trigger
- mag+phase complex splitting 31ms (inherent cost)

Added prioritized improvement suggestions to PERFORMANCE_PLAN.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

is_invalid(): For numeric dtypes (float, complex, int), skip the
expensive a==None comparison (always False for numeric arrays) and
use np.isnan() directly. 44x faster on 963k complex128 arrays
(44.6ms -> 1.0ms), cascading to 2.8x faster datadict_to_meshgrid.

metadataShape: Move the qcodes_shape check from QCAutoPlotMainWindow
into the parent AutoPlotMainWindow.setDefaults(), so both mpl and
pyqtgraph backends benefit. When shape metadata exists, skip the
expensive guess_grid_from_sweep_direction entirely.

Remove now-unused packaging.version import from autoplot.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…date

is_invalid(): Skip a==None for numeric dtypes (always False), use
np.isnan directly. 44.6ms -> 1.0ms on 963k complex128 arrays.
Cascades to 2.8x faster datadict_to_meshgrid (122ms -> 44ms).

mpl double-replot: setData() was triggering _plotData() twice —
once via setAllowedPlotTypes signal and once explicitly. Block
toolbar signals during option updates to eliminate the redundant
plot cycle. Saves ~500ms per setData on 963x1001 data.

label(): Remove validate() call — label() only needs field lookup,
not full monotonicity/shape validation. Saves ~33ms per label on
large MeshgridDataDicts.

metadataShape default: Move qcodes_shape check to parent class
so both mpl and pyqtgraph benefit. Remove dead version import.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Mark is_invalid, metadataShape, and mpl double-replot as done.
Add backend comparison table (mpl vs pyqtgraph after optimizations).
Add artist-level mpl updates as future improvement suggestion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…alidate

Records counter: Count rows from the results table instead of using
result_counter (which counts INSERT calls, not data points for array
paramtype). Falls back to result_counter when results table doesn't
exist (e.g., qdwsdk downloads).

is_invalid(): Skip a==None for numeric dtypes — 44x faster on complex
arrays, cascading to 2.8x faster datadict_to_meshgrid.

mpl double-replot: Block toolbar signals during setData() option
updates to eliminate redundant _plotData() call (~20% faster replot).

label(): Remove validate() call — label only needs field lookup.

metadataShape: Default to metadataShape when qcodes_shape exists,
for both backends. Remove obsolete qcodes version check.

Add 12 regression tests covering axis orientation, records counter,
dataset refresh (incremental + inspector-level), and 1D complex
data splitting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

pyqtgraph imagData: Reset to False before checking data, so switching
from complex to non-complex data correctly disables complex options.

Grid layout: Add equal row/column stretch factors so all grid cells
get equal space, preventing elongated plots when many subplots are
arranged in the grid.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Records counter: Count rows from results table instead of using
result_counter (which counts INSERT calls, not data points for array
paramtype). Falls back to result_counter when table doesn't exist.

Complex mode: Reset imagData flag before checking data so switching
from complex to non-complex correctly updates toolbar options.

Aspect ratio: Add equal row/column stretch factors to pyqtgraph
grid layout so all cells get equal space.

label(): Use .get() with defaults to handle DataDictBase instances
that haven't been validated (no 'label'/'unit' keys yet).

Select All / Deselect / 1D / 2D buttons: New buttons in the data
selector widget. Batch selection (blockSignals) ensures a single
signal emission and single replot per button click. 1D/2D buttons
are only visible when the dataset has dependents with that many axes.

21 regression tests covering axis orientation, records counter,
dataset refresh, 1D complex splitting, and selection buttons.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Recursion fix: Separate setSelectedData (original, per-item signal)
from setBatchSelectedData (new, single signal for batch buttons).
The original path through node signalOption/setOption relies on
_emitGuiChange flag which the batch emit bypassed.

Records counter: Read run_description to extract shapes, compute
record count as product of shape dimensions. Falls back chain:
results table rows -> shape from run_description -> result_counter.

Button layout: Use addLayout instead of addItem to keep buttons
inside the NodeWidget's VBoxLayout (fixes separate window issue).

label(): Use .get() with defaults for missing label/unit keys.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rewrite records counter tests to generate data in tmp_path instead
of referencing local fixture files. Remove device identifiers and
dataset GUIDs from PERFORMANCE_PLAN.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Both backends: setData(None) now clears existing plots instead of
silently returning. Deselect-all produces None from DataSelector,
which now correctly empties the plot area.

pyqtgraph PlotBase: Set minimum size 40x40 to prevent QFont point
size <= 0 warnings when pyqtgraph computes tick labels on
zero-sized widgets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Matplotlib backend: Add colormap combo box to the toolbar. Lists
popular colormaps first (viridis, magma, inferno, etc.), then all
others. Changing the colormap updates matplotlib rcParams and
triggers an immediate replot.

Add 5 tests for pyqtgraph complex mode switching: imagData detection,
all options available, switch-to-real-and-back, separate Re/Im mode,
non-complex shows Real only.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Axis inversion: Transpose z data in pyqtgraph setImage() to match
matplotlib convention. The meshgrid first axis maps to the bottom
(x) label but pyqtgraph ImageItem needs it transposed for correct
display orientation.

Deselect all: Remove the button since pyqtgraph's flowchart does
not propagate empty selection downstream (by design — None return
from process() means 'no change'). The Select All / 1D / 2D buttons
already provide sufficient selection control. deselectAll() now
selects the first dependent to ensure the plot always has data.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Renamed deselectAll to selectFirst — selects only the first
dependent, matching the default behaviour when opening a plot window.
Added as a visible button in the data selector toolbar.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…optimization

# Conflicts:
#	plottr/apps/inspectr.py
#	plottr/data/datadict.py
#	plottr/node/autonode.py
#	plottr/node/scaleunits.py
#	plottr/plot/mpl/autoplot.py
#	test_requirements.txt

Master cleaned up inspectr type:ignore comments, so the per-module
override is no longer needed for that module.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CI installs PyQt5, not PyQt6. Use plottr's Qt abstraction layer
(plottr.QtCore, plottr.QtWidgets) for cross-binding compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>